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APPEAL BRIEF 

Dear Sir: 

Appellant submits this Appeal Brief pursuant to the Notice of Appeal filed February 4, 
2008 and the Notice of Panel Decision from Pre-Appeal Brief Review mailed March 25, 2008. 
The Commissioner is authorized to charge Deposit Account No. 21-0279 in the name of United 
Technologies Corporation, $510.00 for the appeal brief fee. Any additional fees or credits may 
be charged or applied to Deposit Account No. 21-0279 in the name of United Technologies 
Corporation. 

REAL PARTY IN INTEREST 

The real party in interest is United Technologies Corporation, assignee of the present 
invention. 
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RELATED APPEALS AND INTERFERENCES 

There are no prior or pending appeals, interferences or judicial proceedings related to this 
appeal, or which may directly affect or may be directly affected by, or have a bearing on, the 
Board's decision in this appeal. 

STATUS OF CLAIMS 

Claims 1-16 are pending and rejected. Claims 1-16 are appealed. 

STATUS OF AMENDMENTS 

All amendments have been entered. 

SUMMARY OF CLAIMED SUBJECT MATTER 

Designing replacement parts, such as aircraft engine and helicopter airframe sheet metal 
parts, require accurate three-dimensional (3D) CAD models to provide an accurate tool path for 
machining the replacement parts. Ideally, the 3D CAD model precisely defines all the geometric 
elements in a given part, their dimensions and the connectivity relationships between them. 
However, current methods all have inherent deficiencies. 

The disclosed and claimed invention is directed to a 3D modeling method and system that 
automatically creates a full 3D CAD model of regular featured geometry sheet metal items 
quickly and automatically. The modeling system 100 includes a 2D gray- tone vision system 
integrated with a 3D scanning system. The system first generates a two-dimensional (2D) grey- 
tone geometric map of the object. The 3D scanning system then scans the object and fits the 
scanned 3D points to the geometries noted in the 2D map. The camera 102 captures a profile of 
an object 104 to be modeled. The camera 102 sends image data to an image grabber 108 in the 
computer 106, which captures the image taken from the cameras 102 and stores it as a 2D gray- 
tone image. An image processor 110 breaks the 2D gray-tone image from the image grabber 108 
into its component geometric objects. A memory 111 stores a geometric element standards 
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library from which the image processor 110 can choose the geometric elements to match the 2D 
gray-tone image. (Page 4, lines 3-15, paragraph 14, Figure 1) 

The system 100 includes a 3D scanning system that scans the object and generates a 3D 
stereoscopic image (e.g., a point cloud). The 3D scanner includes a laser projector 120 that 
projects a plurality of fine laser stripes onto the object 104. The cameras 102 then capture the 
images of the object 104 illuminated by the projected stripes and send this information to a 
second image processor 122, which generates a 3D stereoscopic image. (Page 4, lines 16-24, 
paragraph 15). 

A method carried out by the system may include a calibration step that calibrates the 
cameras 102 for both 2D vision and 3D stereovision (block 200). After calibration, the cameras 
102 and image grabber 108 capture 2D pixel data, forming a 2D gray-tone image representation 
of the object 104 (block 202). The image processor 110 then breaks down the 2D gray-tone 
image into its geometric elements (block 204) and validates the geometric images against 
geometries in the standard image library (block 206). The geometric element breakdown and 
validation step fits the geometric elements present in the 2D gray-tone image against standard 
geometries in the library, resulting in the 2D geometry model. (Page 5, lines 10-24, Page 6, lines 
1-25, paragraphs 18-23, Figure 2). 

The system 100 uses the 2D geometry model in conjunction with the 3D scanning system 
to generate the final model. Once the scanning scheme has been created (block 208), the image 
processor 110 creates a list of geometric elements present in the 2D geometry model so that it 
can later fit points obtained via the 3D scanning process to match the geometries (block 210). 
Next, the laser projector 120 projects a plurality of stripes onto the object 104 and the cameras 
102 capture 3D stereoscopic images of the portions illuminated by the stripes (block 212). The 
second image processor 122 then calculates the 3D coordinates of the illuminated points using, 
for example, stereoscopic point-matching techniques (i.e., by matching points on each 
illuminated stripe on each camera screen to a corresponding point on the object 104) to generate 
a 3D stereoscopic image (e.g., a point cloud) of the object 104 (block 214). (Page 6, linesl2-24, 
paragraphs 21-22, Figure 2). 
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Once all of the desired 3D stereoscopic images have been obtained, the second image 
processor 122 matches segments of the point clouds obtained via the 3D scanning process with 
the geometric elements obtained in the 2D imaging process (block 218). By starting with a 2D 
geometric element framework to construct a 3D model, the inventive method and system not 
only generates a precise 3D model of a given object, but can actually compensate for distortions 
and imperfections in the original object by fitting the detected 3D stereoscopic image with the 
geometric elements forming the object. Further, treating the object as a combination of geometric 
elements rather than as a sculpted parametric surface also compensates for distortions and 
imperfections while at the same time reducing the processing resources used to generate the 
image. As a result, the invention creates a more precise 3D CAD model, in less time, than 
currently known methods. (Page 7, lines 9-18, paragraphs 23-24, Figure 2). 

Claim 1 

Claim 1 recites a modeling system 10 including at least one camera 102, an image 
grabber 108 that captures a two dimensional (2D) image of an object 104, a scanner 106 that 
scans the object 104 to create a three dimensional (3D) image of the object and at least one 
image processor 110 that breaks the 2D image into geometric elements and matches the scanned 
3D image with the geometric elements to generate the model. (Page 4, lines 3-15, paragraph 14, 
Figure 1) 
Claim 9 

Claim 9 recites a method of generating a model of an object including capturing a two 
dimensional (2D) image of an object 104, scanning the object to create a three dimensional (3D) 
image of the object, breaking the 2D image into geometric elements, and matching the 3D image 
with the geometric elements to generate the model. (Page 6, lines 12-24, paragraphs 21-22, 
Figure 2). 
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GROUNDS OF REJECTION TO BE REVIEWED ON APPEAL 

(1) Is the rejection of Claims 1-3, 5, 7-9, 12-14 under 35 U.S.C. § 103(b) as being 
obvious over U.S. Patent No. 5,821,943 to Shashua ("Shashua") in view of IEEE paper authored 
by Park ("Park") improper? 

(2) Is the rejection of Claim 4 under 35 U.S.C. § 103(b) as being obvious over 
Shashua in view of Park and U.S. Patent No. 5,144,685 to Nasar et al. ("Nasar") improper? 

(3) Is the rejection of Claims 6 and 11 under 35 U.S.C. § 103(b) as being obvious 
over Shashua in view of Park and U.S. Patent No. 5,995,650 to Migdal et al. ("Migdal") 
improper? 

(4) Is the rejection of Claim 15 under 35 U.S.C. § 103(b) as being obvious over 
Shashua in view of Park and U.S. Patent No. 4,638,156 to Horikawa et al. ("Horikawa") 
improper? 

(5) Is the rejection of Claim 16 under 35 U.S.C. § 103(b) as being obvious over 
Shashua in view of Park and Article authored by Fabio Remodndino ("Remodndino") improper? 

ARGUMENT 

(1) Rejection of claims 1-3, 5, 7-9, 12-14 as being obvious over Shashua in view of Park 
Claims 1 and 9 

Claim 1 requires an image processor that breaks a 2D image into geometric elements and 
matches scanned 3D images with the geometric elements to generate a model of an object. Claim 
9 requires the steps of breaking a 2D image into geometric elements and matching a 3D image to 
the geometric elements to generate the model of the object. 
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The proposed combination does not even disclose or suggest all the limitations of claims 
1 and 9. All that Park and Shashua disclose is the creation of a 3D image. However, claims 1 
and 9 require the creation of a 2D image and the creation of a 3D image, and also require the 
additional step of matching the 3D image with geometric elements selected utilizing the 2D 
image. 

The disclosures of Park and Shashua do not disclose and cannot suggest this step and 
feature. Instead, both the Park and Shashua systems and methods disclose a method of generating 
a 3D image using points form various 2D images. Nothing in Park or Shashua disclose breaking 
the 2D image into geometric elements and matching a scanned 3D image with the geometric 
elements. In fact, both Park and Shashua disclose various matrix methods for relating select 
points in each image to all the other images. There is no selection of a geometric element, but 
instead the matching of similar points in each image to provide as close a replica to the original 
as possible. 

Additionally, neither, Park or Shashua disclose matching scanned 3D data to geometric 
elements obtained from a 2D image. Instead, Park and Shashua use the 2D images to create the 
3D image, not as a separate image that is matched with geometric images broken out form a 2D 
image. 

Further, Shashua does not disclose breaking the 2D image into geometric elements as 
argued by the Examiner. The Examiner argues that the term "geometric level" discussed in col 
5, lines 55-60 of Shashua discloses this feature. However, as is defined in Shashua the 
"geometric level" term is merely a relation between locations and features, not a break down of 
the 2D image into geometric elements. This is not the same as the claimed breaking of the 2D 
image into geometric elements. 

For these further reasons, the proposed combination is improper as it fails to disclose or 
suggest all the claimed features and should be withdrawn. 
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Additionally, the Examiner argues that the reasoning supporting the proposed 
combination is that both provide methods of producing 3D images and that the addition of the 
scanner disclosed in Park with the Shasuha device would provide an effective 3D object 
modeling application from accurate and reliable data as suggested by Park. (Final rejection 
10/04/2007, page 3). However this passage in Park simply provides a description of uses of 
structured light scanners, but does not provide any reason as to why anyone would make the 
proposed combination. 

The examiner dismisses the fact that the proposed combination would destroy an intended 
operation and purpose of the base reference (Shashua). The Park system uses a 2D image with 
projected lines this is accomplished with the aid of lines projected onto an object. In the Shashua 
system the 3D image is obtained from several different 2D images. 

Park specifically states that more than one camera for obtaining 2D images is not 
desirable. (Page 66, 2 nd paragraph from bottom in Column 2). Therefore, the Examiner is 
attempting to combine a system that requires and utilizes many 2D images to compile a 3D 
image (Shashua) with a system that uses only one 2D image, and that specifically states that 
more than one 2D camera or image reduces system performance without benefit. The entire 
reason for using the scanned light as disclosed in the Park reference is to obtain 3D images with 
only one 2D image. In contrast, the Shashua device utilizes many 2D images to formulate the 
desired 3D image. For these reasons, the two references teach away from the proposed 
combination. 

Further, the proposed combination would require such a change in operation of the base 
reference (Shashua) as to destroy its intended operation. As appreciated, a proposed 
combination cannot destroy an intended operation of the base reference. In this case, the purpose 
of Shashua is to generate a novel view of a 3D scene. The novel view is not one that is provided 
by any of the 2D images. (Col 1, lines 25-65, and Col 3, lines 10-20). The use of the scanning 
disclosed in the Park reference is utilized to eliminate the need for more than one camera and 
view to reduce the accumulation of errors and other problems encountered with the use of 
multiple cameras and images (Park, page 65, last paragraph, and page 66 first full paragraph). 
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The entire disclosure of the Shashua reference provides a process of combining many 2D images 
from different cameras for creating a 3D image from many 2D images, where the entire 
disclosure of the Park references is focused on using only one camera. 

If the scanning system disclosed in the Park system was somehow combined with the 
Shashua device, the Shashua device would no longer operate as intended and would become 
inoperable for its intended purpose. There can be no reason for making a proposed combination, 
if that combination would destroy operation of the base reference. For at least this reason, the 
proposed combination is improper and this rejection should be reversed. 

Additionally, the disclosures in Shashua teach away from the use of a 3D scanned 
image. As appreciated, the entire puipose of Shashua is to produce a 3D image; there is nothing 
that follows after that creation. In other words, the end product of Shashua and also of the 
disclosed Park method and device is the 3D image. Claims 1 and 9 utilize a 3D image in 
combination with a 2D image to create a model of an object. This step and feature is simply not 
disclosed or suggested by Shashua and Park. For these further reasons the proposed 
combination is not proper and all rejections based on the combination of Shashua and Park 
should be reversed. 

Claim 5 

Claim 5 requires a geometric elements standards library reflecting geometric element 
characteristics, where the image process validates the geometric elements by referencing the 
library. These features are not discloses or suggested by the proposed combination. 

The examiner cites to Shashua (col 10, lines 45-48 and col 13, lines 6-8) as disclosing 
CAD model information. Simply because Shashua mentions a CAD model does not mean that 
the recited features are disclosed or suggested. In this case, the Shashua CAD model is utilized 
to store probe locations, and has nothing to do with a geometric standards library and validation 
of geometric images based on the stored library. Instead, the cited passages in Shashua are part 
of the process of using many points in many images to match multiple 2D images to create a 3D 
image. None of which discloses or suggests the features of claim 5. 
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(2) Rejection of Claim 4 as being obvious over Shashua in view of Park and Nasar 

Claim 4 requires a processor for matching a 3D image with geometric elements of a 2D 
image. For the reasons discussed above with regard to claims 1 and 9, this feature cannot be 
suggested or disclosed by the proposed combination. As best understood, Nasar is added to the 
combination of Shashua and Park because it discloses different processors. The addition of 
Nasar does not correct the deficiencies in the Shashua and Park references, and therefore the 
rejection to claim 4 should be reversed. 

(3) Claims 6 and 11 as being obvious over Shashua in view of Park and Migdal et al. 
Claim 6 requires a correction matrix reflecting distortion between the 3D image and the 

geometric elements. Claim 1 1 requires the steps of determining a difference between the 3D 
image and the geometric elements, generating a correction matrix based on the difference to 
reflect a distortion between the 3D image and the geometric elements and correcting the 3D 
image based on the correction matrix. 

As best understood, Migdal was added to the combination of Shashua and Park to 
disclose a correction matrix. However, Migdal does not disclose a correction matrix as claimed. 
The Migdal matrix is utilized as part of a image collection device calibration device (Col 18, 
lines 10-15), and not as a tool to correct a 3D image in view of geometric elements selected from 
a 2D image. The examiner's rejection requires one to jump from the Migal calibration matrix to 
a process where a geometric element selected form a 2D image is compared to a 3D image for 
correcting the 3D image. These steps and limitations are not suggested by the proposed 
combination. 

(4) Claim 15 as being obvious over Shashua in view of Park and Horikawa et al. 

Claim 15 requires the step of controlling a scanning speed based on information from the 
breaking step. The breaking step referrers to the limitation in claim 9 of breaking the 2D image 
into geometric elements. As discussed above, the combination of Shashua and Park does not 
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disclose breaking a 2D image into geometric elements. All that is discloses the matching of 
many points between different 2D images. Horikawa et al. is added to the above combination to 
disclose adjustment of a scanning speed. However, claim 15 requires that the speed be adjusted 
based on the information for the breaking step. This step is not disclosed or suggested by the 
proposed combination. Accordingly, this rejection should be reversed. 

(5) Claim 16 as being obvious over Shashua in view of Park and Remondino. 

Claim 16 was rejected as being obvious over the combination of Shashua and Park in 
view of the thesis by Remondino. Claim 16 depends from claim 9 that is allowable form for the 
reasons discussed above. The addition of Remondino does not correct the deficiencies in the 
proposed combination. Accordingly, claim 16 is also allowable and this rejection should be 
reversed. 

CONCLUSION 

For the reasons set forth above, the rejection of claims 1-16 is improper and should be 
reversed. Appellant earnestly requests such an action. 

Respectfully Submitted, 

CARLSON, GASKEY & OLDS, P.C. 

/John M. Siragusa/ 

John M. Siragusa 
Registration No. 46,174 
400 West Maple Road, Suite 350 
Birmingham, Michigan 48009 
Telephone: (248) 988-8360 
Dated: April 25, 2008 Facsimile: (248) 988-8363 
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CLAIMS APPENDIX 

1. A modeling system for modeling an object, comprising: 
at least one camera; 

an image grabber that captures a two dimensional (2D) image of the object; 
a scanner that scans the object to create a three dimensional (3D) image of the object; and 
at least one image processor that breaks the 2D image into geometric elements and 
matches the scanned 3D image with the geometric elements to generate the model. 

2. The modeling system of claim 1, wherein the 2D image is a 2D gray- tone image 
and the 3D image is a 3D stereoscopic image. 

3. The modeling system of claim 1, wherein said at least one camera comprises two 
cameras in a spaced spatial relationship. 

4. The modeling system of claim 1, wherein said at least one image processor 
comprises a first image processor that breaks the 2D image into the geometric elements and a 
second image processor that matches the 3D image with the geometric elements. 

5. The modeling system of claim 1, further comprising a memory accessible by said 
at least one processor, wherein the memory stores a geometric element standards library 
reflecting geometric element characteristics and wherein said at least one image processor 
validates the geometric elements by referencing the library. 

6. The modeling system of claim 1, wherein said at least one image processor further 
generates a correction matrix reflecting distortion between the 3D image and the geometric 
elements, wherein said at least one image processor corrects the 3D image based on the 
correction matrix. 
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7. The modeling system of claim 1, wherein the scanner is a laser projector that 
projects a plurality of illuminated stripes on the object. 

8. The modeling system of claim 1, further comprising at least one movable 
platform supporting at least one of said at least one camera and the object to move said at least 
one camera and the object relative to each other. 

9. A method of generating a model of an object, comprising: 
capturing a two dimensional (2D) image of the object; 

scanning the object to create a three dimensional (3D) image of the object; 

breaking the 2D image into geometric elements; and 

matching the 3D image with the geometric elements to generate the model. 

10. The method of claim 9, further comprising validating the geometric elements with 
a geometric element standards library reflecting geometric element characteristics, wherein the 
validating step is conducted before the matching step. 

1 1 . The method of claim 9, further comprising: 

determining a difference between the 3D image and the geometric elements; 
generating a correction matrix based on the difference to reflect any distortion between 
the 3D image and the geometric elements; and 

correcting the 3D image based on the correction matrix. 

12. The method of claim 9, wherein the scanning step comprises projecting a plurality 
of illuminated stripes on the object and capturing at least one image of portions of the object 
illuminated by the stripes. 
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13. The method of claim 12, wherein the projecting step projects the illuminated 
stripes at a first set of locations, and wherein the scanning step further comprises projecting the 
plurality of illuminated stripes on the object at a second set of locations different than the first set 
of locations and capturing at least one image of portions of the object illuminated by the stripes 
in the second set of locations. 

14. The method of claim 9, wherein the 3D image comprises at least one point cloud, 
and wherein the matching step comprises: 

segmenting said at least one point cloud into a plurality of point cloud segments; and 
matching each of the plurality of point cloud segments with one of the geometric 
elements, 

and wherein the method further comprises merging the plurality of point cloud segments 
to generate the model. 

15. The method of claim 9, further comprising controlling a scanning speed based on 
information from the breaking step. 

16. The method of claim 9, wherein the 3D image comprises at least one point cloud, 
and wherein the matching step further comprises at least one of removing outliers and reducing 
point density in said at least one point cloud. 



13 



Serial No. 10/738,403 
67097-033; EH-1 1026 



Evidence Appendix 

None. 
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Related Proceedings Appendix 

None 
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